Last time I changed my desktop computer I bought a CPU from the Intel Haswell family, the one available on the market at that time. I carefully selected the CPU to make sure it supports as many instructions extensions as possible in this family (Intel likes segmentation, even high-end CPUs like the Core i7-4770k do not support all possible instructions). I ended-up choosing the Core i7-4771 as it supports the Transactional Synchronization Extensions (Intel TSX) instructions, which provide transactional memory support. Support for it has been recently added in the GNU libc, and has been activated in Debian. By choosing this CPU, I wanted to be sure that I can debug this support in case of bug report, like for example in bug#751147. Recently some computing websites started to mention that the TSX instructions have bugs on Xeon E3 v3 family (and likely on Core i7-4771 as they share the same silicon and stepping), quoting this Intel document. Indeed one can read on page 49:

HSW136. Software Using Intel TSX May Result in Unpredictable System Behavior

Problem: Under a complex set of internal timing conditions and system events, software using the Intel TSX (Transactional Synchronization Extensions) instructions may result in unpredictable system behavior.
Implication: This erratum may result in unpredictable system behavior.
Workaround: It is possible for the BIOS to contain a workaround for this erratum.

And later on page 51:

Due to Erratum HSw136, TSX instructions are disabled and are only supported for software development. See your Intel representative for details.

The same websites tell that Intel is going to disable the TSX instructions via a microcode update. I hope it won t be the case and that they are going to be able to find a microcode fix. Otherwise it would mean I will have to upgrade my desktop computer earlier than expected. It s a bit expensive to upgrade it every year and that s a the reason why I skipped the Ivy Bridge generation which didn t bring a lot from the instructions point of view. Alternatively I can also skip microcode and BIOS updates, in the hope I won t need another fix from them at some point.

Five years ago Debian and most derivatives switched from the standard GNU C Library (GLIBC) to the Embedded GLIBC (EGLIBC). Debian is now about to take the reverse way switching back to GLIBC, as EGLIBC is now a dead project, the last release being the 2.19 one. At the time of writing the glibc package has been uploaded to experimental and sits in the NEW queue. EGLIBC is dead for a good reason: the GLIBC development has changed a lot in the recent years, due to two major events: Ulrich Drepper leaving Red Hat and the GLIBC development, and the GLIBC steering committe self-dissolving. This has resulted in a much more friendly development based on team work with good cooperation. The development is now based on peer review, which results in less buggy code (humans do make mistakes). It has also resulted in things that were clearly impossible before, like using the same repository for all architectures, and even getting rid of the ports/ directory. Before we used to have two sets of architectures, the main ones in the glibc repository with architectures like x86, SuperH or SPARC, and the secondary ones in the glibc-ports repository with architectures like ARM or MIPS. As you can see the separation was quite arbitrary, and often leaded to missing changes on the secondary architectures. We also got real stable branches, with regular fixes. The most important EGLIBC features have been merged to GLIBC, including for example the use of non-bash but POSIX shell, or the renaming of reserved keywords. The notable exception is the support for configurable components, which we originally planned to use for Debian-Installer, by building a smaller flavor using -Os and without NIS and RPC support. At the end we never worked on that, and it seems that the hardware targeted by Debian has grown faster than the GLIBC size, so that is not really a big loss. At the end, we ended up with only 5 missing patches from the EGLIBC tree:

• Installation of *_pic.a files for use of mklibs in debian-installer
• Dynamic reload of /etc/resolv.conf
• Installation of headers during bootstrapping
• Workarounds for PowerPC 8xx CPUs
• Access to FPSCR on SH4

The package names are unchanged (except the source package and the binary package containing the sources) so the transition is fully transparent for the users. I would like to thank all the CodeSourcery employees who worked on EGLIBC, with a special thank to Joseph Myers who spent countless hours to merge the most important EGLIBC changes back to GLIBC, and sent regular emails about the merge status. I would also like to thanks all the people on the GLIBC side that made the change to happen, and all persons participating in the GLIBC development.

I will never understand the point of using autotools, cmake or whatever configure system, when later the code uses an hardcoded list of architectures to determine the size of a pointer Unfortunately for porters this pattern is quite common. Update: As people keep asking, the way to check for the size of a given type is explained in the autoconf manual. To check for the size of the pointer, the following entry has to be added to configure.ac: AC_CHECK_SIZEOF(void *) On a 64-bit system, this will lead to the following entry in config.h:

/* The size of void *', as computed by sizeof. */
#define SIZEOF_VOID_P 8


Following the release of Debian Wheezy, I have (finally) updated my set of Debian QEMU images for both Squeeze (6.0.8) and Wheezy (7.3). The following images are now available:

• amd64 (Squeeze and Wheezy)
• armel (Squeeze and Wheezy)
• armhf (Wheezy)
• i386 (Squeeze and Wheezy)
• mips (Squeeze and Wheezy)
• mipsel (Squeeze and Wheezy)
• powerpc (Squeeze and Wheezy)

Each of these directories contains a GPG signed README.txt file with the md5sums all files, detailed instructions how to run these images and especially the minimum QEMU version to use. The requirements to run the default desktop environment have increased a lot between Squeeze and Wheezy. An accelerated graphics card is now needed to be able to use Gnome (unless you use the fallback mode), which is not something provided by QEMU (actually there is now a QXL para-virtualized video card, but the driver is only in unstable). In addition GDM now needs more than 128MiB to start, while this is the default amount of memory provided for virtual machines. I have therefore decided to switch the default desktop environment on the Wheezy images to Xfce and the display manager to LightDM. Both Gnome and GDM are still installed on the images, and the original default can easily be restored using the following commands:

• update-alternatives --auto x-session-manager
• echo /usr/sbin/gdm3 > /etc/X11/default-display-manager

Beside this the new images only contain minor changes. The filesystems have been tweaked to not run fsck after a certain amount of days, and locales-all and openssh-server are now installed in all images. For MIPS and MIPSEL, 64-bit kernels are now also installed and provided, so that it is possible to choose between a 32-bit or a 64-bit kernel (see the README.txt for more details). There is no Debian Wheezy SPARC image available, as QEMU does not fully support SPARC64 yet (it is actually possible to run it, but then the VM crashes often), and Debian Wheezy now only supports 64-bit kernels. I will also invest time to build an S390X image, but so far I haven t been successful on that. The following images are still available at the same location, though they haven t been updated:

• sparc (Etch)
• SH4 (Sid from a few years ago)

I read a lot this year - I'll write more about that and reflections on goodreads in another post - but most of the things I read weren't published in 2013. (I should also write a bit about my thoughts on e-readers). However, it seems I have enough to write about 2013's novels to make a round-up post worthwhile, so here we go. This year, crime author Robert Galbraith published his first novel The Cuckoo's Calling. I'd never have heard of it if Galbraith was not outed as an alias for Joanne "JK" Rowling. Clues that Rowling was working on a detective story exist as early as a Guardian preview article in 2012 for her last novel, The Casual Vacancy. Further hints, for me, that this was no first-time author were the taglines from Ian Rankin and Val McDermid on the cover, writers of a calibre I'd be surprised a new author could attract. However I don't know whether they were on the pre-unveiling cover or not. Rowling was upset be outed, having enjoyed the freedom to write without the baggage of expectation that she is subject to. I hope she's pleased: prior to her unmasking the novel was warmly received by the (admittedly relatively small) number of people who read it. And a very good novel it is too. It starts with a genre clich of a grizzled, meloncholy detective, Mr. Cormoran Strike, in an upstairs office with a neon light flickering through the window, but fleshes the story out both forwards - a client, a mysterious death - and backwards - how did Mr. Strike end up in that upstairs office - living out of it, no less? As is traditional for the genre there's a very clever twist. What I really enjoyed about Cormoran Strike was Galbraith/Rowling moving quickly from Chandler-esque everyman to a well fleshed-out, complex protagonist, intertwining the development of the character with the unfolding of the wider plot. I'm looking forward to the sequel, expected in 2014. A second surprise favourite this year was Lauren Beukes' time-tripping crime story The Shining Girls. A monsterous murder of women somehow finds a room in Chicago that lets him travel through time (or perhaps the room finds him). He uses this facility to stalk and murder a set of Shining Girls: women who, for one reason or another, literally 'shine' in his perception of them. One such woman survives his first attack and decides to try and find out who attacked her, and why. The crimes are described in a brutal fashion which - from a distance - resemble the sometimes glorified violence for which crime fiction is sometimes criticised, but the focus of the story is very much on the victims: they are fully fleshed out characters and each death is felt by the reader as a genuine tragedy. I discovered Beukes when her earlier novel Zoo City was included in a Humble eBook bundle. On reading The Shining Girls I felt that the novel deserved to be more widely known than I would expect it to be trapped in the ghetto of genre fiction, so I was pleased to discover that the very mainstream Richard and Judy Book Club discovered it. In established author news, Terry Pratchett, having adopted speech recognition for writing (to combat his debilitating Alzheimer's) has seemingly managed to accelerate his rate of production and squeezed out at least two this year: The Long War with Stephen Baxter is the sequel to 2012's The Long Earth which I very much enjoyed, but it really felt like "difficult second novel" to me. Hopefully there'll be a third. Raising Steam, the 40th Discworld novel, was an enjoyable romp around the concept of steam trains, featuring the relatively new Moist von Lipwig who has managed to become one of my favourite Discworld characters. I can't think of much more to say about the novel, really. It's a Discworld novel, probably not the best introduction to the series for a new reader, but will give a reader familiar with the franchise everything they expect, and possibly no more. Iain Banks sadly died this year, shortly after the publication of his last novel, The Quarry. It's sat on my hardback shelf for the time being. I couldn't bring myself to read it in 2013. I did read his last SF offering from the year prior, The Hydrogen Sonata. Sadly, yet coincidentally, both of these books examine the nature of living and dying, The Quarry in particular from the point of view of a terminal cancer sufferer. I have a small backlog of unread Banks fiction which I want to take my time over with. Finally, whilst not really a book, I thoroughly enjoyed the BBC's 2013 adaptation of Neil Gaiman's Neverwhere. Natalie Dormer wrote a piece on the making of the drama which should serve as a good introduction. At the time of writing, most of the programmes have disappeared from iPlayer, but I would be surprised if this wasn't released commercially at some point.

Shanley wrote an insightful article about microagressions and management [1]. It s interesting to read that and think of past work experiences, even the best managers do it. Bill Stone gave an inspiring TED talk about exploring huge caves, autonamous probes to explore underground lakes (which can be used on Europa) and building a refuelling station on the Moon [2]. Simon Lewis gave an interesting TED talk about consciousness and the technology needed to help him recover from injuries sustained in a serious car crash [3]. Paul Wayper wrote an interesting article about reforming the patent system [4]. He also notes that the patent system is claimed to be protecting the mythical home inventor when it s really about patent trolls (and ex-inventors who work for them). This is similar to the way that ex-musicians work for organisations that promote extreme copyright legislation. Amanda Palmer gave an interesting TED talk about asking for donations/assistance, and the interactions between musicians and the audience [5]. Some part of this are NSFW. Hans Rakers wrote a useful post about how to solve a Dovecot problem with too many files open [6]. His solution was for a Red Hat based system, for Debian you can do the same but by editing /etc/init.d/dovecot. The use of the /proc/N/limits file was interesting, I ve never had a cause to deliberately use that file before. Krebs on Security has an interesting article about Android malware being used to defeat SMS systems to prevent bank fraud [7]. Apparently an infected PC will instruct the user to install an Android app to complete the process. Rick Falkvinge wrote an interesting article about how to apply basic economics terminology to so-called Intellectual Property [8]. Matthew Garrett wrote an interesting post about the way that Ubuntu gets a better result than Debian and Fedora because it has clear fixed goals [9]. He states that many people regard Fedora as a playground to produce a range of niche derivatives , probably a large portion of the Fedora and Debian developers consider this a feature not a bug. Ming Thein wrote an interesting article about the demise of the DSLR [10]. Bruce Schneier wrote an interesting post on the detention of David Miranda by the British authorities [11]. It s mostly speculation as to why they would do such a thing (which seems to go against their own best interests) and whether the NSA even knows which documents Edward Snowden copied. Jaclyn Friedman wrote an interesting article on Mens Rights Movements (MRAs) and how they are bad for MEN as well as for women [12]. Rodney S. Tucker wrote an insightful article for the IEEE about the NBN [13]. Basically the Liberal party are going to spend most of the tax money needed for a full NBN but get a significantly less than the full benefit. Lauren Drell wrote an interesting article for Mashable about TellSpec, a portable spectrometer that communicates with an Android phone to analyse food for allergens [14]. I guess this will stop schools from banning phones. Katie McDonough wrote an interesting article for Salon about the Pope s statements about the problems with unchecked capitalism [15]. His ideas are really nothing new to anyone who has read the Bible and read the news. It seems to me that the most newsworthy part of this is that most Christian leaders don t make similar statements. Daniel Leidert wrote an interesting post about power saving when running Debian on a HP Microserver [16]. Most of it is relevant to other AMD64 hardware too, I ll have to investigate the PCIE ASPM and spin down options on some of my systems that are mostly idle.

• [1] https://medium.com/about-work/65d4740f7a2f
• [2] http://www.ted.com/talks/bill_stone_explores_the_earth_and_space.html
• [3] http://tinyurl.com/6zgcb3f
• [4] http://mabula.net/tbfw/2013/08/01
• [5] http://www.ted.com/talks/amanda_palmer_the_art_of_asking.html
• [6] http://tinyurl.com/k3xz9a8
• [7] http://krebsonsecurity.com/2013/08/a-closer-look-perkele-android-malware-kit/
• [8] http://tinyurl.com/orf9gds
• [9] http://mjg59.dreamwidth.org/27517.html
• [10] http://blog.mingthein.com/2013/08/24/the-demise-of-the-dslr/
• [11] https://www.schneier.com/blog/archives/2013/08/detaining_david.html
• [12] http://prospect.org/article/good-mens-rights-movement-hard-find
• [13] http://tinyurl.com/q49b9fa
• [14] http://mashable.com/2013/11/26/tellspec/
• [15] http://tinyurl.com/nosxkj2
• [16] http://www.wgdd.de/2013/08/hp-n54l-microserver-energy-efficiency.html

I wanted to create a Deb-o-Matic environment to testbuild packages for a different architecture. Taking inspiration on St phane s excellent blog post, I tried to replicate the creation of a cross-architecture Linux container in Debian. Here are the steps I made: Load binfmt_misc module:
# modprobe binfmt_misc Install the required packages:
# apt-get install lxc debootstrap rsync qemu-user-static binfmt-support Mount the cgroup virtual filesystem:
# mkdir /cgroup
# echo "none /cgroup cgroup defaults 0 0" >> /etc/fstab
# mount -a I needed a specific template to create a cross-architecture container. I used the excellent one by Laurent Vivier. Download it, rename it as lxc-cross-debian, mark it executable, and store it under /usr/share/lxc/templates. Create the cross-architecture container, an armhf one in this case:
# lxc-create -t cross-debian -n debian-armhf -- --arch armhf --suite sid --interpreter-path /usr/bin/qemu-arm-static After a while, the container was created and I enjoyed my brand new armhf test machine

Sometimes building the same code on multiple architectures is useful to detect horrors like:

ExEnv::err0() < < sprintf("AtomInfo: invalid name: %s\n",name.c_str());
This code builds using -Werror=format-security with only a few warnings on most architectures, while GCC is correctly detects the issue on some others. This has been reported as bug#728249.

Some catchup blogging: Kurt Elling was back in town on October 19, and not just in Chicago but quite literally in our little 'burb as the headliner for this season's concerts at Domiminican University. They had done us the favour of inviting Dianne Reeves back in March 2009, but this was clearly going to be a real treat. And I managed to snatch four front row (!!) tickets as soon as I heard about it, and the show once again didn't disappoint. Elling was as usual accompanied by his long-time collaborator Laurence Hobgood on piano, as well as local heros Clark Sommers on bass and John McLean on guitar. This time, Quincy Davies was on drums. And this band is realiably excellent, as is Elling in any live setting. The set was dominated by pieces from his most recent record and augmented by a few other standout pieces. There is a pretty rich set of live music by Elling on YouTube, see for example On Broadway (which was part of the set in a wonderfully fast and funked-up variant), Golden Lady (from the previous album, and also played live this time) or for example this wonderful version of You Send Me (also part of the concert). I snatched a photo or two (while at least turning my flash off, unlike the weird gal to my left, but I digress) and posted one on Google+. Kurt Elling will be back in Chicago at the Green Mill in early January. Expect to me there, and I hope to see you too!

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

During the last week I was attending the first Heidelberg Laureates Forum as one of the lucky 200 accepted young scientists. The HFL is a (from now on hopefully yearly) event that brings together Fields Medalists, Abel Prize laureates and Turing Award winners with young scientists (undergraduates, Ph.D. students and postdocs) from both fields in the city of Heidelberg. The extremely well organized week consisted of lectures from the laureates, some workshops held by postdocs, excursions and plenty of good food. Videos of the lectures are available (but don t work on Linux, at least not for me ), and I have shot a few pictures of the event as well. I believe that my favourite lectures where Michael Atiyah s Advice to a Young Mathematician , Vladimir Voevodsky s Univalent Foundations of Mathematics , William Morton Kahan s Desperately Needed Remedies for the Undebuggability of Large-Scale Floating-Point Computations in Science and Engineering and Alan Kay s Putting Turing to Work . Where are all the functional programmers? During that event, one gets to talk to many other math and computer scientists researchers; sometimes just Where are you from? and What do you do? , sometimes long discussions. Unfortunately, I hardly found one who is into functional programming language research is that only because the event was parallel to ICFP (which I really would have liked to attend as well), or is functional programming really just about 1% of all computer science? What is a proof? My other research interest lies in interactive theorem proving, especially using Isabelle. Of course that is a topic that one can discuss with almost everyone at that event, including the mathematicians. The reactions were rather mixed: On the one end of the spectrum, some mathematicians seriously doubt that they would ever trust a computer to check proofs and that it would ever be efficient enough to use. Others would not mind having a button that tells whether their paper written in LaTeX is correct , but were not keen to invest time or thought into making the proof readable by the computer. And then there were some (but very few!) who had not heard of theorem proving before and were very excited by the prospect of being able to obtain certainty about their proofs immediately and without having to bother other scientists with it. During the mathematician s panel discussions, where I posed the question Do you see value in or even a need for machine-checked proofs in mathematics. , Efim Zelmanov (Fields Medal 1994) said a proof is what other mathematicians see as a proof . I found this attitude a bit surprising for me, a proof has always been a rigorous derivation within a formal system (say, ZFC set theory), and what we write in papers is a (less formal) description of the actual proof, whose existence we believe in. Therefore I was very happy to see Vladimir Voevodsky give a very committed talk about Univalent Foundations and how using that as the language for mathematics will allow more of mathematics to be cast in a formal, machine-checked form. I got the chance to discuss this with him in person, as I wanted to hear his option on Isabelle, and especially on the usefulness of the style of structured proof that Isar provides, and which is closer to the style of proofs that mathematicians use in papers. He said that he enjoyed writing his proof in the style required in Type Theory and in Coq, and that maybe mathematicians should and will adjust to the language of the system, while I believe that a structured proof languages like Isar, independent of the underlying logic (HOL in this case; which is insufficient to form a base for all of abstract mathematics), is a very useful feature and that proof assistants should adjust to the mathematicians.
We also briefly discussed the idea of mine to work with theorem provers already with motivated students in high schools, e.g. in math clubs, and found that simple proofs about arithmetic of natural numbers could be feasible here, without being too trivial. All in all a very rewarding and special week, and I can only recommend to try to attend one of the next forums, if possible.

During DebConf, Asheesh presented the idea of using git instead of the file system for storing the contents of Debian Code Search. The hope was that it would lead to fewer disk seeks and less data due to gits delta-encoding. Maybe the reduction would be big enough that enough data could be held in RAM to allow for fast retrieval. Joey Hess helped me out with a couple of details: he revealed that using git repack -a -d will lead to a single packfile that optimally contains HEAD, not caring about the history. Also, he showed me how to use git cat-file. We did a small-scale experiment and the results were promising. I told him I ll do the experiment of just committing the entire unpacked source mirror to git and promised to follow up with the results, so here goes:

stapelberg@couper ~/unpacked master $ time cat <<EOT   git cat-file --batch >/dev/null
:linux_3.2.32-1/sound/pci/ice1712/aureon.c
:linux_3.2.32-1/sound/soc/codecs/sgtl5000.c
:linux_3.2.32-1/sound/pci/hda/patch_conexant.c
EOT
cat <<<''  0,00s user 0,00s system 62% cpu 0,006 total
git cat-file --batch > /dev/null  4,38s user 1,08s system 99% cpu 5,477 total
stapelberg@couper ~/unpacked master $ time cat linux_3.2.32-1/sound/pci/ice1712/aureon.c linux_3.2.32-1/sound/soc/codecs/sgtl5000.c linux_3.2.32-1/sound/pci/hda/patch_conexant.c >/dev/null
cat linux_3.2.32-1/sound/pci/ice1712/aureon.c   > /dev/null  0,00s user 0,00s system 69% cpu 0,006 total
stapelberg@couper ~/unpacked master $ time cat <<EOT   git cat-file --batch >/dev/null
:i3-wm_4.6-1/libi3/font.c
EOT
cat <<<':i3-wm_4.6-1/libi3/font.c'  0,00s user 0,00s system 51% cpu 0,008 total
git cat-file --batch > /dev/null  4,30s user 1,18s system 99% cpu 5,533 total
stapelberg@couper ~/unpacked master $ time cat i3-wm_4.6-1/libi3/font.c >/dev/null
cat i3-wm_4.6-1/libi3/font.c > /dev/null  0,00s user 0,00s system 0% cpu 0,004 total

Even after repeating those tests a couple of times to get a warm page cache, the result stays the same: git takes about 5 seconds to resolve the deltas in a large repository. Even if this was 5 seconds startup time and a very small amount of additional time per file, it would not be acceptable for our use case. The conclusion is that git is clearly not suitable for this kind of usage, which is not surprising after having heard a couple of times that git does not scale :-). For the curious, the .git/objects directory is 29 GiB for roughly 140 GiB of source code, so the delta encoding is quite impressive in terms of saving space. However, keep in mind that the compressed (!) Debian source archive is about 35 GiB, so the savings are not that huge.

Laurent Bigonville reported Debian bug #720000 on Saturday August 17th 2013, against the drizzle package. This bug is already marked pending by Tobias Frost, the package maintainer. Bug #710000 was reported as of May 27th: 2 months and 21 days for 10,000 bugs. For once, this is a rate acceleration which we can probably explain by the release of wheezy and the work strongly resumed by many maintainers for the release of jessie. It is indeed interesting to see that this 720000th bug report happened nearly on Debian's 20th birthday. To make it short, we could then say that Debian had 36,000 bug reports every year in average (which is not exactly true as the BTS records start in 1996). Funnily also, this is the first time since I'm doing this recurrent post every 10,000 bugs that one happens *during* a DebConf, a few hours before DebConf 13 officially ends up.

As you might (or might not) know, this summer, I have taken on mentoring of a GSoC project by Simon Chopin (a.k.a. laarmen) which goal is to bring fedmsg, the Fedora Infrastructure message bus, to Debian. Most of the work I ll be talking about here is Simon s work, please send all the praise towards him (I can take the blame, though). What is this about? As the project proposal states, the idea is to provide Debian with a unified, real-time, and open mechanism of communication between its services. This communication bus would allow anyone, anywhere, to start consuming messages and reacting to events happening in Debian s infrastructure:

• trigger a test build on a git push to a source repository
• trigger automated testing (piuparts, lintian) as soon as an upload hits the archive
• get a desktop notification when a package you care about gets changed

When we told upstream about our plan of adapting fedmsg to work on Debian, they were thrilled. And they have been very supportive of the project. How is the project going? Are you excited? I know I m excited. Well, the general idea was easy enough, but the task at hand is a challenge. First of all, fedmsg has a lot of (smallish) dependencies, most of them new to Debian. Thanks to Simon s work during the bonding period, and thanks to paultag s careful reviews, the first batch of packages (the first dependency level, comprising kitchen, bunch, m2ext, grapefruit, txws, txzmq and stomper) is currently sitting in the NEW queue. The four remaining packages (fabulous, moksha.common, moksha.hub and fedmsg proper) are mostly ready, waiting in the Debian Python Module Team SVN repository for a review and sponsorship. While we re waiting for the packages to trickle into Debian, Simon is not twiddling his thumbs. Work has taken place on a few fronts:

• Getting fedmsg integrated to mentors.debian.net (actually sending the first messages from Debian s infrastructure)
• Writing a (stop-gap) wrapper to convert some of our mail broadcasts (debian-devel-changes, debian-bugs-dist) into fedmsg messages, until a more proper integration can be done
• Working on reliability of the message transport, following some concerns raised by DSA.

fedmsging mentors.debian.net Package backports mentors.debian.net was chosen because I m an admin and could do the integration quickly. That involved backporting the eleven aforementioned packages, plus zeromq3 and python-zmq (that only have TCP_KEEPALIVE on recent versions), to wheezy, as that s what the mentors.d.n host is running. (Also, python-zmq needs a new-ish cython to build so I had to backport that too). Thankfully, those were no-changes backports, that were easily scripted, using a pbuilder hook to allow the packages to depend on previously built packages. I have made a wheezy package repository available here. It s signed with my GnuPG key, ID 0xB8E5087766475AAF, which should be fairly well connected. Code changes After Simon s initial setup of debexpo (which is not an easy task), the code changes have been fairly simple (yes, this is just a proof of concept). You can see them on top of the live branch on debexpo s sources. I finally had the time to make them live earlier this week, and mentors.debian.net has been sending messages on Debian s fedmsg bus ever since. Deployment mentors.d.n sends its messages on five endpoints, tcp://mentors.debian.net:3000 through tcp://mentors.debian.net:3004. That is one endpoint per WSGI worker, plus one for the importer process(es). You can tap in directly, by following the instructions below. debmessenger Debmessenger is the stop-gap email-to-fedmsg bridge that Simon is developing. The goal is to create some activity on the bus without disrupting or modifying any infrastructure service. It s written in hy, and it leverages the existing Debian-related python modules to do its work, using inotify to react when a mail gets dropped in a Maildir. Right now, it s supposed to understand changes mails (received from debian-devel-changes) and bugs mail (from debian-bugs-dist). I ll work on deploying an instance of debmessenger this weekend, to create some more traffic on the bus. Reliability of the bus I suggested using fedmsg as this was something that already existed, and that solved a problem identical to the one we wanted to tackle (open interconnection of a distribution s infrastructure services). Reusing a piece of infrastructure that already works in another distro means that we can share tools, share ideas, and come up with solutions that we might not have considered when working alone. The drawback is that we have to either adapt to the tool s idiosyncrasies, or to adapt the tool to our way of working. One of the main points raised by DSA when the idea of using fedmsg was brought up, was that of reliability. Debian s infrastructure is spread in datacenters (and basements ) all over the world, and thus faces different challenges than Fedora s infrastructure, which is more tightly integrated. Therefore, we have to ensure that a critical consumer (say, a buildd) doesn t miss any message it would need for its operation (say, that a package got accepted). There has been work upstream, to ensure that fedmsg doesn t lose messages, but we need to take extra steps to make sure that a given consumer can replay the messages it has missed, should the need arise. Simon has started a discussion on the upstream mailing list, and is working on a prototype replay mechanism. Obviously, we need to test scenarios of endpoints dropping off the grid, hence the work on getting some activity on the bus. How can I take a look? a.k.a. Another one rides the bus (Picture Yves-Laurent Allaert, CC-By-SA v2.5 / GFDL v1.2 license) So, the bus is pretty quiet right now, as only two kinds of events are triggering messages: a new upload to mentors.debian.net, and a new comment on a package there. Don t expect a lot of traffic. However, generating some traffic is easy enough: just login to mentors.d.n, pick a package of mine (not much choice there), or a real package you want to review, and leave a comment. poof, a message appears. For the lazy Join #debian-fedmsg on OFTC, and look for messages from the debmsg bot. Current example output:

01:30:25 <debmsg> debexpo.voms-api-java.upload (unsigned) --
02:03:16 <debmsg> debexpo.ocamlbricks.comment (unsigned) --

(definitely needs some work, but it s a start) Listening in by yourself You need to setup fedmsg. I have a repository of wheezy packages and one of sid packages, signed with my GnuPG key, ID 0xB8E5087766475AAF. You can add them to a file in /etc/apt/sources.list.d like this:

deb http://perso.crans.org/dandrimont/fedmsg-<sid wheezy>/ ./

Then, import my GnuPG key into apt (apt-key add), update your sources (apt-get update), and install fedmsg (apt-get install python-fedmsg). The versions are << to anything real, so you should get the real thing as soon as it hits the archive. Finally, in /etc/fedmsg.d/endpoints.py, you can comment-out the Fedora entries, and add a Debian entry like this:

    "debian": [
        "tcp://fedmsg.olasd.eu:9940",
    ],

fedmsg.olasd.eu runs a fedmsg gateway connected to the mentors.d.n endpoints, and thus forwards all the mentors messages. It ll be connected to debmessenger as soon as it s running too. To actually see mesages, disable validate_signatures in /etc/fedmsg.d/ssl.py, setting it to False. The Debian messages aren t signed yet (it s on the roadmap), and we don t ship the Fedora certificates so we can t authenticate their messages either. Finally, you can run fedmsg-tail --really-pretty in a terminal. As soon as there s some activity, you should get that kind of output (color omitted):

 
  "i": 1, 
  "msg":  
    "version": "2.0.9-1.1", 
    "uploader": "Emmanuel Bourg <ebourg@apache.org>"
   , 
  "topic": "org.debian.dev.debexpo.voms-api-java.upload", 
  "username": "expo", 
  "timestamp": 1373758221.491809
 

Enjoy real-time updates from your favorite piece of infrastructure! What s next? While Simon continues working on reliability, and gets started on message signing according to his schedule, I ll take a look at deploying the debmessenger bridge, and making the pretty-printer outputs useful for our topics. There will likely be some changes to the messages sent by debexpo, as we got some feedback from the upstream developers about making them work in the fedmsg tool ecosystem (datanommer and datagrepper come to mind). You can tune in to Simon s weekly reports on the soc-coordination list, and look at the discussions with upstream on the fedora messaging-sig list. You can also catch us on IRC, #debian-soc on OFTC. We re also hanging out on the upstream channel, #fedora-apps on freenode.

Dear project members, here's another report of DPL activities, this time for a period longer than usual (February + 1st week of March), so that the next one will be at the very end of the current DPL term. Highlights

• As you know, we are now well within the DPL election process. And we have 3 valuable candidates running. I encourage all of you to participate in the discussions on -vote, and ask questions about project vision, goals, and improvements. It's something that is rarely as intense as during campaigning, so don't miss the chance!
• You might have noticed a sharp decrease in the count of RC bugs affecting Wheezy; that is largely due to the Release Team, who started the usual final sweep of RC triaging before release. Please thank them for their work, ... but don't give up on our collective work yet! You can still help up in the usual ways (NMUs, severity adjustments, unblock reviews, etc).
• The technical committee had to deal with a rather urgent issue during last month (#699808). I mention it here only to applaud their efficiency in doing so: 4 days to reach a decision. Resorting to tech-ctte shall always remain a last resort in Debian, but when it comes to that it's useful to know that we can count on a wise and speedy answer.

Appointments

• As the previous term was about to end, I've agreed with the Kurt Roeckx to re-appoint him as Project Secretary for another year.
• Following up to last month news, we've now assembled a team of admins for Debian participation into Google Summer of Code 2013 and delegated them for the task. Many thanks to David, Nicolas, Paul, and Sukhbir for volunteering, as well as to last year admins for their help in reaching out to interested volunteers. On a related note, we've until March 28th to propose projects and/or volunteer as (co-)mentor for GSoC 2013.

DPL helpers Two more DPL helpers IRC meetings have happened, minutes and logs of both are available. Assets

• I've worked with the press and publicity teams to announce the new trademark policy more widely and call for producing Wheezy merchandise. Apparently, we are now also being cited as a reference on how to strike a balance between free software and trademark (unfortunately the article is behind a paywall now).
• In related news, Brian Gupta has volunteered to help with answering trademark@d.o inquiries and has already helped a lot in streamlining the process and keeping track of past requests (thanks!)
• Following a -project inquiry by Thomas Koch, I've investigated with SPI the possibility of assigning to them the copyright for (code) contributions to Debian. The bottom line is that at present there is no safe way to do that with SPI; that might change in the future.
• As agreed at a DPL helpers meeting, we are trying to federate interests in Debian sponsoring and fund-raising. Part of the goal is avoid duplication of efforts DebConf- and sponsoring for other Debian activities; and part is establishing a more stable income flow for Debian (to ease long-term budget decisions). If you have experience and interest in this area, please join the debian-sponsors-discuss list on Alioth.
• Together with auditors, we have updated reimbursement procedures to better keep track of both outstanding and past requests. Main difference is that you'll now have to mail a RT queue instead of leader@d.o directly. Check the wiki page for details.

Events Past At the beginning of February, I've attended FOSDEM 2013, together with many other Debian people. I didn't have any specific talk this year, but it's been a chance to talk F2F about several ongoing issues (see logs), and help mediating in some conflicts. I've also accepted the invitation to participate in the GNOME Advisory Board meeting, together with Laurent Bigonville of our GNOME team. No report of that has been prepared as of yet (sorry about that), but we have both reported "live" to the rest of the team on IRC. Future

• It looks like my last month as DPL will be quite busy. Next week I'll be first in New York City, delivering an invited Debian talk at NYLUG (thanks a lot to Brian Gupta and Tom Limoncelli for the invitation). Then I'll head to LibrePlanet 2013 to talk about the relationship between Debian and GNU (thanks to John Sullivan for the invitation). Finally, at the beginning of April, I'll be in Amsterdam to deliver a talk about Debian experience with various legal issues across the years, at the yearly FSFE Legal and Licensing Workshop (thanks to Karsten Gerloff for the invitation). Both trips (LibrePlanet and FSFE) will be on Debian budget. While I usually insist on having travel sponsorship from inviting entities, in this cases I've accepted to do otherwise given they are free software non profits like Debian.
• I've been invited to represent Debian at Distro Recipes. Due to a conflict with FSFE workshop I couldn't make, so I've looked for substitutes. Lucas Nussbaum and Jonas Smedegaard have kindly accepted to go in my stead and deliver two talks, one about QA and the other about Pure Blends; thanks folks!

Miscellaneous A couple of months ago I've mentioned that I had filed an application, as Debian representative, to participate in a working table to define software procurement rules for the Italian public administration. Good news: my application has been accepted, together with those of other well-known FOSS communities and organizations (e.g. KDE, FSFE). I'll keep you posted of how it goes. Let's go back to elect a new DPL and release Wheezy now,
Cheers.

---

PS the day-to-day activity logs for February and March 2013 are available at the usual place master:/srv/leader/news/bits-from-the-DPL.txt.20130 2,3

Kurt Elling is back in Chicago for two gigs at the Green Mill (where I last saw him in a wonderful big-band setting), along with his long-time collaborator Laurence Hobgood on piano, Clark Sommers on bass, Chicago's own John McLean on guitar, and the new kid Bryan Carter on drums. And they were, of course, awesome. Great mix of standards as well as new stuff from the Grammy-nominated new recording. If you can see Kurt Elling live, go. Now. They return for another gig tonight and I will have to see if I can swing that.

This is my monthly summary of my free software related activities. If you re among the people who made a donation to support my work (120.46 , thanks everybody!), then you can learn how I spent your money. Otherwise it s just an interesting status update on my various projects. Dpkg At the start of the month, I reconfigured dpkg s git repository to use KGB instead of the discontinued CIA to send out commit notices to IRC (on #debian-dpkg on OFTC, aka irc.debian.org). I didn t do anything else that affects dpkg and I must say that Guillem does not make it easy for others to get involved. He keeps all his work hidden in his private for 1.17.x branch and refuses to open an official jessie branch as can be seen from the lack of answer to this mail. On the bright side, he deals with almost all incoming bugs even before I have a chance to take care of them. But it s a pity that I can never review any of his fixes because they are usually pushed shortly before an upload. Misc packaging I helped to get #689336 fixed so that the initrd properly setups the keymap before asking for a passphrase for an encrypted partition. Related to this I filed #689722 so that cryptsetup gains a dependency ensuring that the required tools for keymap setup are available. I packaged a new upstream version of zim (0.57) and also a security update for python-django that affected both Squeeze and Wheezy. I uploaded an NMU of revelation (0.4.13-1.2) so that it doesn t get dropped from Wheezy (it was on the release team list of leaf packages that would be removed if unfixed) since my wife is using it to store her passwords. I sponsored a new upstream version of ledgersmb. Debian France We managed to elect new officers for Debian France. I m taking over the role of president, Sylveste Ledru is the new treasurer and Julien Danjou is the new secretary. Thank you very much to the former officers: Carl Chenet, Aur lien Jarno and Julien Cristau. We re in the process of managing this transition which will be completed during the next mini-Debconf in Paris so that we can exchange some papers and the like. In the first tasks that I have set myself, there s recruiting two new members for the boards of directors since we re only 7 and there are 9 seats. I made a call for volunteers and we have two volunteers. If you want to get involved and help Debian France, please candidate by answering that message as soon as possible. The Debian Handbook I merged the translations contributed on debian.weblate.org (which led me to file this wishlist bug on Weblate itself) and I fixed a number of small issues that had been reported. I made an upload to Debian to incorporate all those fixes But this is still the book covering Squeeze so I started to plan the work to update it for Wheezy and with Roland we have decided who is going to take care of updating each chapter. Librement Progress is annoyingly slow on this project. Handling money for others is highly regulated, at least in the EU apparently. I only wanted an escrow account to secure the money of users of the service but opening this account requires either to be certified as a payment institution by the Autorit de contr le prudentiel or to get an exemption from the same authority (covering only some special cases) or to sign a partnership with an established payment institution. Being certified is out of scope for now since it requires a minimum of 125000 EUR in capital (which I don t have). My bank can t sign the kind of partnership that I would need. So I have to investigate whether I can make it fit in the limited cases of exemption or I need to find another payment institution that is willing to work with me. Gittip uses Balanced a payment service specialized in market places but unfortunately it s US-only if you want to withdraw money from the system. I would love a similar service in Europe If I can t position Librement as a market place for the free software world (and save each contributor the hassle to open a merchant account), then I shall fallback to the solution where Librement only provides the infrastructure but no account, and developers who want to collect donations will have to use either Paypal or any other supported merchant account to collect funds. That s why my latest spec updates concerning the donation service and the payment service mentions Paypal and the possibility of choosing your payment service for your donation form. Thanks See you next month for a new summary of my activities.

5 comments Liked this article? Click here. My blog is Flattr-enabled.

It's been some time since I didn't write in this blog about my running activities. So let's make an update for my international friends. If you don't care about running, you can move to the next post in your feed reader..:-) After a quite busy running activity during DebConf 12 in Nicaragua, as well as during the touristic trip we did afterwards, I went back to my "regular" schedule. Main objectives in September-December are roughly the following:

• Seine-Eure marathon, the flattest marathon in France, on October 21st. This marathon is the one where I set the record which I just broke in June, in Caen. So, the goal is easy to find : break it again. My target time is around 3h37. Not so ambitious in apparence, with a current record at 3h38'45". However, I can't afford the highly meticulous preparation I did in April-May-June, so I try to stay with an achievable target.
• The Le Raid Le Puy-Firminy 70km night run on November 24th. Here, I'll just try to go faster than 9h15 which I achieved last year, trying to have a better race management and avoid cramps. Of course, meteorological conditions will be the key, here, as it can be as low as -15 C in that part of France, in November..:-)

During August, I broke my monthly distance record by running nearly 300km in one month, even achieving 100km in only 5 days, around 24th. That was one of my goals: improve my overall resistance to run repetitions, which is one of the keys for ultra-running. These were achieved with many runs to/back work, with up to 13km in the morning and the same in the evening. Yes, running 26km at about 11km/h (and with some ups and downs in the woods to make it even worse) on a day where one has a normal work activity is quite a challenge. September has been more focused on marathon preparation. This time, no complicated plan with interval running as the month was also a very busy work month, where fitting training sessions during the day would be hard. So, I made an easy plan : run a half-marathon every Sunday and run part of my work-home commute every day (So, 3km run to the train station, 30mn train ride, then 4km run to my work place, then shower....and the same back after work). Out of the 5 half-marathons I ran during these 5 Sundays, two of them were official ones. On the first one, I finally managed to break my personal best on half, with 1h37'14". Only a 14 seconds improvement, but that one is certainly one of my best among personal bests....even better than 3h38'45" on marathon.. All other half-marathons were run at marathon speed, so targeting 5'06"/km, which will be my planned pace on October 21st. The one I ran on Sept. 16th, which was the other official race I ran was finally done quite significantly faster than this. First of all, because I had hard times to run "only" at 5'06"/km because of other runners emulation. And also, because I ran the last 3 kilometers up to nearly 14km/h (so, down to 4'15"/km), just for fun, because I could do it..:-) With all this preparation, I think that I now manage to very well manage my marathon speed. I'll probably do a final test tomorrow by trying to keep running at this speed for 3 rounds of my favourite "Maurepas Marathon" circuit, which is just exactly 1/4 of a marathon. More "funnily", I also did something I never did before during this really crazy month : simply said, I came back from work by running. All the way long. Through woods, forests, along some lakes and finally in the country. 42 kilometers (yes, a marathon). After a work day. Starting at 5:45pm and arriving home after 4h50 minutes. With 2 hours of heavy rain. With 2.5 hours running in the dark (with my headlamp of course). All alone. That was a crazy bet to do....but really great fun achieving this : the GPS trace is here. Really something I have to do again..:-) So, well, now I'm more or less prepared and having fun is just a matter for time..:-). I'll keep you guys posted with those and, guess what? I'm already picking my target races for next year..:-)

The other day, I found myself confronted with a large number of large files. Which were presented in (gzip-)compressed ascii format---which R reads directly via gzfile() connections---as well as (compressed) numpy files. The numpy can be read very efficiently into Python. We can do the same in R via save() and load(), of course. But the trouble is that you need to read them first. And reading hundreds of megabytes from ascii is slow, no matter which language you use. Concerning R, I poked aound scan(), played with the colClasses argument and looked at the recent LaF package written just for this purpose. And all these solutions were still orders of magnitude slower than reading numpy. Which is no surprise as it is really hard to beat binary formats when you have to parse countless ascii tokens. So the obvious next idea was to read the numpy file in Python, and to write a simple binary format. One helpful feature with this data set was that it contained only regular (rectangular) matrices of floats. So we could just store two integers for the dimensions, followed by the total data in either one large binary blob, or a sequence of column vectors. But one minor trouble was that the Intertubes lead to no easy solution to unpack the numpy format. StackOverflow had plenty of question around this topic converned with, say, how to serialize in language-independent way. But no converters. And nobody local knew how to undo the "pickle" format underlying numpy. But a remote friend did: Laurent, well-known for his Rpy2 package, pointed me towards using the struct module and steered me towards the solution shown below. So a shameless plug: if you need a very experienced Python or R consultant for sciece work, consider his consulting firm. Finally, to round out this post, let's show the simple solution we crafted so that the next guy searching the Intertubes will have an easier. Let us start with a minimal Python program writing numpy data to disk:

#!/usr/bin/env python
#
# simple example for creating numpy data to demonstrate converter
import numpy as np
# simple float array
a = np.arange(15).reshape(3,5) * 1.1
outfile = "/tmp/data.npy"
np.save(outfile, a)

Next, the simple Python converter to create a binary file containing two integers for row and column dimension, followed by row times columns of floats:

#!/usr/bin/python
#
# read a numpy file, and write a simple binary file containing
#   two integers 'n' and 'k' for rows and columns
#   n times k floats with the actual matrix
# which can be read by any application or language that can read binary
 
import struct
import numpy as np
inputfile = "/tmp/data.npy"
outputfile = "/tmp/data.bin"
# load from the file
mat = np.load(inputfile)
# create a binary file
binfile = file(outputfile, 'wb')
# and write out two integers with the row and column dimension
header = struct.pack('2I', mat.shape[0], mat.shape[1])
binfile.write(header)
# then loop over columns and write each
for i in range(mat.shape[1]):
    data = struct.pack('%id' % mat.shape[0], *mat[:,i])
    binfile.write(data)
binfile.close()

Lastly, a quick littler script showing how R can read the data in a handful of lines:

#!/usr/bin/r
infile <- "/tmp/data.bin"
con <- file(infile, "rb")
dim <- readBin(con, "integer", 2)
Mat <- matrix( readBin(con, "numeric", prod(dim)), dim[1], dim[2])
close(con)
print(Mat)

That did the job---and I already used to converter to read a few weeks worth of data for further analysis in R. This obviously isn't the last word on possible solutions as the additional temporary file can be wasteful (unless it forms a cache for data read multiple times). If someone has nice solutions, please don't hold back and contact me. Thanks again to Laurent for the winning suggestion concerning struct, and help in getting the examples shown here to work.

Yay, so we made it: s390x got added as a release architecture. What this means:

• If the package already exists on that architecture and fails to build, that issue is deemed RC. You can debug the issue on the porterbox (zelenka), chroot sid_s390x. Build dependencies are installed by a team, just follow the request guidelines.
• If your package is not installable on s390x, it will not migrate to testing anymore. So if a package does not exist on s390x, you need to make sure not to generate a dependency on it or prevent your package from building on s390x (by e.g. build-depending on a package that's not available on s390x or by getting it into Packages-arch-specific).

This will also help other 64bit big-endian ports (like powerpc64 and sparc64) to enter the archive more easily, as most issues left are indeed related to endianness, not to specialities of the System z hardware.

Many thanks go to Aur lien Jarno, without whom this would not have been possible. I also want to take this opportunity to thank all our s390(x) machine sponsors: ZIVIT, IIC@KIT and Marist College. There are not many mainframe owners who let free software projects work on their machines.